feat(data-retention): granular PII redaction stages (input + block outputs) by TheodoreSpeaks · Pull Request #5272 · simstudioai/sim

TheodoreSpeaks · 2026-06-29T22:20:09Z

Summary

Add two execution-altering PII redaction stages alongside the existing log redaction: redact the workflow input before execution, and mask every block output in-flight before the next block reads it
Per-stage policy (entity types + language) for each of Logs / Workflow input / Block outputs; resolved most-specific-wins per workspace, with full back-compat for existing logs-only rules
In-flight stages fail-fast (abort the run) on a Presidio error instead of scrubbing or leaking; the logs stage keeps scrub-to-marker
Reuse the shared HTTP → Presidio path; block-output redaction runs before payload compaction so offloaded large values are still masked
Settings UI: chip-tabs across the three stages, language-first picker with the entity grid filtered to that language's recognizers, and a confirmation before removing a workspace override

Type of Change

New feature

Testing

Tested manually. Unit tests for resolver back-compat, redactObjectStrings + failure modes, and the contract schema. bun run lint, check:api-validation:strict, and check:migrations origin/staging all pass.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…tputs)

vercel · 2026-06-29T22:20:15Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Jun 30, 2026 7:46pm

cursor · 2026-06-29T22:20:19Z

PR Summary

High Risk
Changes execution-time data (inputs, block outputs, streams, memory) and log persistence with fail-fast vs scrub semantics; misconfiguration or Presidio outages can abort runs or affect workflow correctness, not only observability.

Overview
Adds three independently configurable PII redaction stages (workflow input, block outputs, logs), each with its own entity types and language, while legacy flat rules still map to logs-only.

Runtime: Workflow input is masked before execution when the input stage is on. Block outputs are masked in-flight (before compaction and downstream blocks), including buffer-only streaming so raw chunks are not forwarded when that stage is enabled; policy propagates to child workflows and agent memory writes. Input/block stages abort the run on Presidio failure (onFailure: 'throw'); log persistence keeps scrub-to-marker behavior. Log redaction now uses only the logs stage, applies without the pii-redaction feature flag (stored rules are the source of truth), and hydrates large-value refs before masking so offloaded content gets the logs policy.

Presidio / batching: New /analyze_batch and /anonymize_batch endpoints; masking paths chunk by shared byte/count budgets and use batched analyze/anonymize instead of per-string concurrency.

Settings & contracts: Data retention UI uses stage tabs, language-filtered entity grids, and confirm-on-remove for overrides; API/schema accept stages with validation (enabled stages must pick at least one entity type).

^{Reviewed by Cursor Bugbot for commit f0c71cc. Bugbot is set up for automated code reviews on this repo. Configure here.}

greptile-apps · 2026-06-29T22:25:29Z

Greptile Summary

This PR adds stage-based PII redaction for workflow execution and settings. The main changes are:

Separate policies for workflow input, block outputs, and logs.
Execution-time masking for workflow inputs and block outputs.
Batched Presidio analyze/anonymize endpoints and shared masking helpers.
Settings UI updates for per-stage language and entity selection.
Contract and resolver updates for staged PII policies.

Confidence Score: 4/5

This is close, but the restore path should be fixed before merging.

Stored rules now appear to be resolved during execution without depending on the feature flag.
Empty enabled stages are rejected at the API boundary.
Restored large-value refs can still expose raw block-output data after redaction is enabled.

apps/sim/lib/workflows/executor/execution-core.ts

Security Review

Block-output redaction can still miss raw PII stored behind old large-value refs during resume or run-from-block restore.

Important Files Changed

Filename	Overview
apps/sim/lib/workflows/executor/execution-core.ts	Adds input and restored-output masking, but restored large-value refs can still bypass block-output redaction.
apps/sim/lib/billing/retention.ts	Resolves staged PII policies and keeps legacy rules mapped to logs-only behavior.
apps/sim/lib/api/contracts/primitives.ts	Adds staged PII contract validation and rejects enabled stages without selected entity types.
apps/sim/executor/execution/block-executor.ts	Masks block outputs before compaction and avoids forwarding raw streaming chunks when block-output redaction is active.

_{Reviews (13): Last reviewed commit: "fix(data-retention): always apply logs p..." | Re-trigger Greptile}

…flows

TheodoreSpeaks · 2026-06-30T00:40:25Z

@greptile review

… resume

TheodoreSpeaks · 2026-06-30T00:46:12Z

@greptile review

…nconditionally (no fail-open)

TheodoreSpeaks · 2026-06-30T00:57:36Z

@greptile review

…edaction

TheodoreSpeaks · 2026-06-30T01:09:03Z

@greptile review

…redaction # Conflicts: # apps/sim/ee/data-retention/components/data-retention-settings.tsx

TheodoreSpeaks · 2026-06-30T17:23:38Z

@greptile review

…no fail-open)

TheodoreSpeaks · 2026-06-30T17:34:59Z

@greptile review

… = off everywhere

…block-output redaction is off

TheodoreSpeaks · 2026-06-30T19:21:49Z

@greptile review

greptile-apps · 2026-06-30T19:24:03Z

+      // Limitation: this walks inline strings only — values offloaded to
+      // large-value storage are still refs here and are not re-masked. In the
+      // normal flow that is safe (a run with the stage on masks before offload);
+      // the gap is the narrow case of a run that offloaded a large value while
+      // the stage was OFF and is resumed after the stage is turned ON.


Large values bypass masking

When block-output redaction is enabled after a workflow already offloaded large block outputs, this restore path only masks inline strings in the snapshot. The offloaded payloads stay behind large-value refs. On resume or run-from-block, downstream blocks can still read the raw restored payload, and log persistence can skip the large-value scrub because block-output redaction is now enabled. This leaves raw PII reachable from prior block outputs after the stage is turned on.

cursor · 2026-06-30T19:25:51Z

          abortSignal: ctx.abortSignal,
+          // Propagate in-flight block-output redaction into child workflows so
+          // nested blocks mask outputs too (recurses: each child forwards it).
+          piiBlockOutputRedaction: ctx.piiBlockOutputRedaction,


Child workflows skip input redaction

Medium Severity

The new workflow-input PII stage runs only in executeWorkflowCore on top-level processedInput. Nested child runs are started with a direct Executor and pass childWorkflowInput unchanged. Only the block-output policy is forwarded on the context, so when the input stage is on and block outputs are off, mapped or explicit child input can execute and produce downstream state without in-flight input masking.

^{Reviewed by Cursor Bugbot for commit 8f86d77. Configure here.}

…logs (preserve redacted content)

TheodoreSpeaks · 2026-06-30T19:34:42Z

@greptile review

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6e9587a. Configure here.}

greptile-apps · 2026-06-30T19:39:55Z

+        snapshot.state.blockStates = await redactObjectStrings(
+          snapshot.state.blockStates,
+          blockOutputOpts
+        )


Refs stay unmasked When a paused run or run-from-block snapshot contains a large-value ref that was created before block-output redaction was enabled, this call only masks inline strings. Large-value refs are treated as opaque by redactObjectStrings, so the ref still points at the original offloaded bytes. The later warm-up step can materialize that raw value for downstream blocks, letting them read or send unredacted PII even though the block-output stage is enabled.

…n logs stage is on

TheodoreSpeaks · 2026-06-30T19:46:37Z

@greptile review

greptile-apps · 2026-06-30T19:49:02Z

+        snapshot.state.blockStates = await redactObjectStrings(
+          snapshot.state.blockStates,
+          blockOutputOpts
+        )


Refs stay raw

This restore path still only masks inline strings. When a paused run or run-from-block snapshot contains a large-value ref created before block-output redaction was enabled, redactObjectStrings leaves the ref untouched. The later warm-up can materialize that original offloaded value for downstream blocks, so the resumed workflow can read raw PII even though block-output redaction is now enabled. This path needs to hydrate, mask, and re-store restored refs before downstream state can use them.

greptile-apps · 2026-07-01T05:46:36Z

+        snapshot.state.blockStates = await redactObjectStrings(
+          snapshot.state.blockStates,
+          blockOutputOpts
+        )
+      }


Large refs remain raw

This restore path still leaves old offloaded block outputs unmasked. It only runs redactObjectStrings over restored blockStates, and that redactor treats large-value refs as opaque, so a paused run or run-from-block snapshot created before block-output redaction was enabled can still point at raw stored bytes. When the restored state is warmed and downstream blocks read that ref, they can receive the original PII even though the block-output stage is enabled. The restore path needs to hydrate, mask, and re-store those refs before exposing the state to execution.

feat(data-retention): granular PII redaction stages (input + block ou…

f24d8cc

…tputs)

cursor Bot reviewed Jun 29, 2026

View reviewed changes

Comment thread apps/sim/executor/execution/block-executor.ts

vercel Bot deployed to Preview June 29, 2026 22:24 View deployment

greptile-apps Bot reviewed Jun 29, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/executor/execution-core.ts Outdated

fix(data-retention): propagate block-output redaction into child work…

36f2a3d

…flows

vercel Bot temporarily deployed to Preview June 30, 2026 00:37 Inactive

cursor Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/executor/execution/block-executor.ts

Comment thread apps/sim/lib/workflows/executor/execution-core.ts

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/executor/execution-core.ts Outdated

fix(data-retention): close block-output redaction gaps on streaming +…

bb3a84b

… resume

vercel Bot temporarily deployed to Preview June 30, 2026 00:45 Inactive

cursor Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/executor/execution/block-executor.ts Outdated

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/executor/execution-core.ts Outdated

fix(data-retention): drain+mask streamed output, resolve PII policy u…

0b81fed

…nconditionally (no fail-open)

vercel Bot temporarily deployed to Preview June 30, 2026 00:56 Inactive

test(testing): support leftJoin().where().limit() in shared db mock

2d75987

vercel Bot temporarily deployed to Preview June 30, 2026 01:05 Inactive

fix(data-retention): mask agent/Pi memory writes under block-output r…

eb6b25a

…edaction

vercel Bot temporarily deployed to Preview June 30, 2026 01:08 Inactive

Merge remote-tracking branch 'origin/staging' into feat/pii-granular-…

324b04c

…redaction # Conflicts: # apps/sim/ee/data-retention/components/data-retention-settings.tsx

cursor Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/app/api/organizations/[id]/data-retention/route.ts

vercel Bot deployed to Preview June 30, 2026 17:01 View deployment

fix(data-retention): guard partial PII stages in GET normalize

d55b557

vercel Bot temporarily deployed to Preview June 30, 2026 17:06 Inactive

vercel Bot temporarily deployed to Preview June 30, 2026 17:23 Inactive

fix(data-retention): enabled stage with no entity types redacts all (…

31f2e3f

…no fail-open)

vercel Bot temporarily deployed to Preview June 30, 2026 17:34 Inactive

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/lib/billing/retention.ts Outdated

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/ee/data-retention/components/data-retention-settings.tsx

cursor Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/executor/execution-core.ts

fix(data-retention): reject enabled stage with no entity types; empty…

437d2bb

… = off everywhere

vercel Bot temporarily deployed to Preview June 30, 2026 17:47 Inactive

docs(data-retention): note resume remask covers inline values only

78b2c56

vercel Bot temporarily deployed to Preview June 30, 2026 17:49 Inactive

fix(data-retention): scrub offloaded large-value refs from logs when …

8f86d77

…block-output redaction is off

vercel Bot temporarily deployed to Preview June 30, 2026 19:21 Inactive

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

cursor Bot reviewed Jun 30, 2026

View reviewed changes

fix(data-retention): hydrate, mask, and re-store large-value refs in …

6e9587a

…logs (preserve redacted content)

vercel Bot temporarily deployed to Preview June 30, 2026 19:34 Inactive

cursor Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/lib/logs/execution/logger.ts Outdated

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

fix(data-retention): always apply logs policy to large-value refs whe…

f0c71cc

…n logs stage is on

vercel Bot temporarily deployed to Preview June 30, 2026 19:46 Inactive

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

waleedlatif1 deleted the branch staging July 1, 2026 05:43

waleedlatif1 closed this Jul 1, 2026

waleedlatif1 reopened this Jul 1, 2026

greptile-apps Bot reviewed Jul 1, 2026

View reviewed changes

Uh oh!

Conversation

TheodoreSpeaks commented Jun 29, 2026

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

Uh oh!

greptile-apps Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Security Review

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

Uh oh!

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

greptile-apps Bot Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jun 30, 2026

Choose a reason for hiding this comment

Child workflows skip input redaction

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

greptile-apps Bot Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

greptile-apps Bot Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vercel Bot commented Jun 29, 2026 •

edited

Loading

cursor Bot commented Jun 29, 2026 •

edited

Loading

greptile-apps Bot commented Jun 29, 2026 •

edited

Loading